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Amendments to the Claims: 

This listing of claims will replace all prior versions, and listings of claims in the application: 
Listing of Claims; 

1 1. (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display[[,]] the multimedia information comprising 

3 information of a plurality of tj i ^^os including information of a first tj i ^po and information of a 

4 second type , the method comprising: 

5 displaying a graphical user interface (GUI) on the display; 

6 displaying, in a first area of the GUI, a roprosontation of the muhimodia 

7 information stored by the multimedia document, the displayed roprosontation of the multimedia 

8 information comprising a r e pr e s e ntation of information of th e first typ e and a r e pr e s e ntation of 

9 information of the second type; 

10 displaying, in a first area of the GUI, a first visual representation of the 

11 multimedia information stored in the multimedia document, the first visual representation 

12 including a first representation of information of a first type stored in the multimedia document 

13 and a first representation of information of a second type stored in the multimedia document: 

1 4 displayin g, in the first area of the GUI, a first lens moveable in response to user 

15 input over representations of multimedia information displayed in the GUI positionable over a 

16 plurality of portions of the first visual representation displayed within the first area of the GUI . 

1 7 the first lens covering a first portion of the first visual representation within the first area; and 

18 displaying, in a second area of the GUI, a representation of multimedia 

19 information displayed in the first portion of the first area, the roprosontation of multimedia 

20 information display e d in th e s e cond ar e a comprising a portion of th e r e pr e s e ntation of 

21 information of th e first typ e cov e r e d by th e first l e ns and a portion of th e r e pr e s e ntation of 

22 information of the second type covered by the first lens 
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23 displaying, in a second area of the GUI, a second visual representation of the 

24 multimedia information stored in the multimedia document based on the first lens covering the 

25 first portion of the first visual representation within the first area, the second visual 

26 representation including a second representation of the information of the first type stored in the 

27 multimedia document and a second representation of the information of the second type stored in 

28 the multimedia document . 



1 2. (Currently amended) The method of claim 1 wherein displaying the first 

2 visual representation of the multimedia information stored [[by]] in the multimedia document in 

3 the first area of the GUI comprises: 

4 displaying a first thumbnail image in the first area of the GUI, the first thumbnail 

5 image comprising the first representation of tiie information of the first type; and 

6 displaying a second thumbnail image in the first area of the GUI, the second 

7 thumbnail image comprising the first representation of tiie information of the second type. 

1 3. (Currently amended) The method of claim 1 wherein displaying, in the 

2 s e cond ar e a of th e GUI, the representation of multim e dia information display e d in th e first 

3 portion of the first area displaying the second visual representation of the multimedia 

4 information stored in the multimedia document comprises: 

5 displayin g, in a first sub-area of the second area of the GUI, the second 

6 representation of the information of the first type as a portion of the first representation of tiie 

7 information of the first type covered by the first lens in a first pan e l in th e s e cond ar e a of th e 

8 GUI; and 

9 displaying , in a second sub-area of the second area of the GUI, the second 

1 0 representation of the information of the first type as a portion of the first representation of the 

1 1 information of the second type covered by the first lens in a s e cond pan e l in th e s e cond ar e a of 

12 t he GUI 

1 4. (Currently amended) The method of claim 1 wherein displaying, in the 

2 second area of the GUI, the representation of multimedia information displayed in the first 



Page 3 of 69 



Appl. No. 10/081,129 PATENT 



Amdt. dated September 29, 2008 
Reply to Office Action of April 28, 2008 

3 portion of tho first aroa displaying the second visual representation of the multimedia 

4 information stored in the multimedia document comprises: 

5 determining a first time and a second time associated with the first lens; 

6 displaying, in the second area of the GUI, a representation of the information of 

7 the first type occurring between the first time and the second time associated with the first lens as 

8 the second representation of the information of the first type ; and 

9 displaying, in the second area of the GUI, a representation of the information of 

1 0 the second type occurring between the first time and the second time associated with the first 

1 1 lens as the second representation of the information of the second type . 

1 5. (Currently amended) The method of claim 1 further comprising: 

2 receiving user input moving the first lens over the first visual representation 

3 displaved within the first area to cover a second portion of the first visual representation within 

4 the first area; and 

5 responsive to the user input, automatically changing the information second visual 



6 representation displayed in the second area of the GUI such that the second visual representation 

7 of the multimedia information stored in the multimedia document displayed in the second area of 

8 the GUI corresponds to the second portion of the first visual representation of tiie multimedia 

9 information stored in the multimedia document covered by the first lens included in tho second 
10 portion of tho first area . 



1 6. (Currently amended) The method of claim 1 fiirther comprising: 

2 displayin g, in the second area of the GUI, a second lens mov e abl e in r e sponse to 

3 user input over roprcsontations of multimedia information displayed in tho GUI positionable over 

4 a plurality of portions of the second visual representation displayed within the second area of the 

5 GUI , the second lens covering a first portion of the second visual representation within the 

6 second area; and 

7 displaying, in a third area of the GUI, a third visual representation of tiie 

8 multimedia information stored in the multimedia document based on the second lens covering 

9 the first portion of the second visual representation within the second area corresponding to th e 
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1 0 first portion of tho second aroa , the third visual representation of multimedia information 

1 1 displayod in tho third area comprising including a portion of tho third representation of tiie 

1 2 information of the first type cov e r e d by th e s e cond l e ns and a portion of th e third representation 

13 of tiie information of the second type cov e r e d by th e s e cond l e ns . 



1 7. (Currently amended) The method of claim 6 wherein displaying, in the 

2 third area of the GUI, the third visual representation of tiie multimedia information stored in the 

3 multimedia document corresponding to the First portion of the second area comprises: 

4 determining a first time and a second time associated with the second lens; 

5 displaying, in the third area of the GUI, a representation of the information of the 

6 first type occurring between the first time and the second time associated with the second lens as 

7 the third representation of the information of the first type : and 

8 displaying, in the third area of the GUI, a representation of tiie information of the 

9 second type occurring between the first time and the second time associated with the second lens 
10 as the third representation of the information of the second type . 

1 8. (Currently amended) The method of claim 6 wherein: 

2 displaying the first visual representation of the multimedia information stored 

3 [[by]] in the multimedia document in the first area of the GUI comprises: 

4 displaying a first thumbnail image in the first area of the GUI, the first 

5 thumbnail image comprising the first representation of tiie information of the first type; and 

6 displaying a second thumbnail image in the first area of the GUI, the 

7 second thumbnail image comprising the first representation of tiie information of the second 

8 type; 

9 displaying the second visual representation of the multimedia information stored 

1 0 in the multimedia document display e d in th e first portion of th e first ar e a in the second area of 

1 1 the GUI comprises: 

1 2 displayin g, in a first sub-area of the second area of the GUI, the portion of 

1 3 the first representation of tiie information of the first type covered by the first lens as the second 
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14 representation of the information of the first type in a first panel in tho second aroa of tho GUI ; 

15 and 

1 6 displayin g, in a second sub-area of the second area of the GUI, the portion 

17 of the first representation of the information of the second type covered by the first lens as the 

18 second representation of the information of the second type in a second panel in the second area 

19 of tho GUI ; and 

20 displaying the third visual representation of the multimedia information stored in 

2 1 the multimedia document corr e sponding to th e first portion of th e s e cond ar e a in the third area of 

22 the GUI comprises: 

23 displayin g, in a first sub-area of the third area of the GUI, the portion of 

24 the second representation of the information of the first type covered by the second lens as the 

25 third representation of the information of the first type corresponding to the first portion of the 

26 second area of tho GUI in a first sub area of the third area of the GUI ; and 

27 displayin g, in a second sub-area of the third area of the GUI, the portion of 

28 the second representation of tiie information of the second type covered by the second lens as the 

29 third representation of the information of the first type corr e sponding to th e first portion of th e 

30 second area of tho GUI in a second sub - area of tho third area of tho GUI . 

1 9. (Currently amended) The method of claim 6 fiirther comprising: 

2 receiving [[a]] user input moving the second lens over the second visual 

3 representation displayed within the second area to cover a second portion of the second visual 

4 representation within the second area; and 

5 responsive to the user input, automatically changing the information third visual 

6 representation displayed in the third area of the GUI such that the third visual representation of 

7 tiie multimedia information stored in the multimedia document displayed in the third area of the 

8 GUI corresponds to the second portion of the second visual representation of the multimedia 

9 information stored in the multimedia document covered by the second lens included in tho 
10 second portion of tho second area . 

1 10. (Currently amended) The method of claim 6 fiirther comprising: 
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2 



receiving [[a]] user input moving the first lens over the first visual representation 



3 displayed within the first area to cover a second portion of the first visual representation within 

4 first area; and 



7 second area of the GUI such that the second visual representation of the multimedia information 

8 stored in the multimedia document displayed in the second area of the GUI corresponds to the 

9 second portion of the first visual representation of tiie multimedia information stored in the 

10 multimedia document covered by the first lens included in the second portion of the first area ; 

11 and 

12 changing the information third visual representation displayed in the third 

13 area of the GUI such that the third visual representation of the multimedia information stored in 

14 the multimedia document displayed in the third area of the GUI corresponds to the second visual 

1 5 representation of the multimedia information stored by the multimedia document within included 

16 in th e s e cond portion of the second area. 

1 11. (Currently amended) The method of claim 6 further comprising: 

2 displaying a sub-lens covering a portion of the first visual representation 

3 displayed within the first area of the GUI corresponding to the first portion of the second visual 

4 representation within the second area of the GUI covered by the second lens. 

1 12. (Currently amended) The method of claim 1 1 fiirther comprising: 

2 receiving [[a]] user input moving the second lens over the second visual 

3 representation displayed within the second area to cover a second portion of the second visual 

4 representation within the second area; and 

5 responsive to the user input, automatically changing [[a]] position of the sub-lens 

6 to cover a portion of the first visual representation displaved within the first area of the GUI 

7 corresponding to the second portion of the second visual representation within the second area 

8 covered by the second lens . 



5 



responsive to the user input, automatically: 



6 



changing the information second visual representation displayed in the 
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1 13. (Currently amended) The method of claim 1 wherein: 

2 the information of the first type corresponds to video information; and 

3 the first representation of the information of the first type comprises one or more 

4 video keyframes extracted from the video information. 

1 14. (Currently amended) The method of claim 13 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the first representation of tiie information of the second type comprises text 

4 information obtained from transcribing the audio information. 

1 15. (Currently amended) The method of claim 13 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the first representation of the information of the second type comprises text 

5 information included in the CC text information. 

1 16. (Currently amended) The method of claim 1 further comprising: 

2 receiving information indicating a user-specified concept of interest; and 

3 analyzing the multimedia information stored in the multimedia document to 

4 identify one or more locations in the multimedia information that are relevant to the user- 

5 specified concept of interest; 

6 wherein display! n g . in the first area of the GUI, the first visual representation of 

7 the multimedia information stored in the multimedia document in th e first ar e a of th e GUI 

8 comprises annotating the one or more locations in the multimedia information that are relevant to 

9 the user-specified concept of interest; and 

10 wherein displaying, in the second area of the GUI, [[a]] the second visual 

1 1 representation of tiie multimedia information stored in the multimedia document display e d in th e 

12 first portion of th e first ar e a comprises aimotating the one or more locations in the multimedia 
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1 3 information that are relevant to the user-specified concept of interest and that are located in the 

1 4 first portion of the first visual representation covered by the first lens within the first area. 

1 17. (Original) The method of claim 1 fiirther comprising: 

2 receiving input indicating selection of a portion of the multimedia information 

3 occurring between a first time and a second time; and 

4 performing a first operation on the portion of the multimedia information 

5 occurring between a first time and a second time. 

1 18. (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a first typo and information of a second typo, the method comprising: 

4 displaying a graphical user interface (GUI) on the display; 

5 displaying, in a first area of the GUI, a representation of the multimedia 



6 information stored [[by]] m the multimedia document occurring between a start time (tg) and an 

7 end time (tg) associated with the multimedia document, the display e d representation of the 

8 multimedia information stored in the multimedia document occurring between tg and tg 

9 comprising a representation of information of [[the]] a first type stored in the multimedia 

1 0 document occurring between tg and tg and a representation of information of [[the]] a second 

1 1 type stored in the multimedia document occurring between t^ and tg, where (tg > t^); 

1 2 displayin g, in the first area of the GUI, a first lens moveable in r e spons e to us e r 

13 input over roprosontations of multimedia information displayed in the GUI positionable over a 

14 plurality of positions within the first area of the GUI, the first lens visually emphasizing a portion 

15 of the first area of the GUI covered by the first lens , the portion of the first area visually 

16 emphasized by the first lens comprising a representation of multimedia information stored in the 

17 multimedia document occurring between a first time (tj) and a second time (t2), where (tg < t j < 

18 t2<tg);and 
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19 displaying, in a second area of the GUI, the representation of tiie multimedia 

20 information stored in the multimedia document occurring between tj and t2 based on the first 

2 1 lens visually emphasizing the portion of the first area , the representation of the multimedia 

22 information stored in the multimedia document occurring between t ] and t ? displayed in the 

23 second area comprising a representation of information of the first type occurring between tj and 

24 t2 and a representation of information of the second type occurring between tj and t2. 

1 19. (Currently amended) The method of claim 18 fiirther comprising: 

2 displayin g, in the second area of the GUI, a second lens moveable in response to 

3 user input over roprosontations of multimedia information displayed in the GUI positionable over 



4 a plurality of positions within the second area of the GUI , the second lens visually emphasizing a 

5 portion of the second area of the GUI covered by the second lens , the portion of the second area 

6 visually emphasized by the second lens comprising a representation of multimedia information 

7 stored in the multimedia document occurring between a third time (t3) and a fourth time (t4), 

8 where (tj < t3 < t4 < t2); and 

9 displaying, in a third area of the GUI, the representation of the multimedia 

1 0 information stored in the multimedia document occurring between t3 and 14 based on the second 

1 1 lens visually emphasizing the portion of the second area , the representation of tiie multimedia 

12 information stored in the multimedia document occurring between U and 1 4 displayed in the 

13 third area comprising a representation of information of the first type occurring between t3 and 

1 4 t4 and a representation of information of the second type occurring between t3 and t4. 

1 20. (Currently amended) The method of claim 19 fiirther comprising: 

2 changing [[the]] position of the first lens in response to user input such that the 

3 first lens visually emphasizes a portion of the first area of the GUI comprising a representation of 

4 the multimedia information stored in the multimedia document occurring between a fifth time 

5 (15) and a sixth time (tg), where (t^ < t5 < tg < tg), (t5 ^ tj), and (tg ^ t2); and 
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6 responsive to the change in the position of the first lens, automatically displaying, 

7 in the second area of the GUI, the representation of tiie multimedia information stored in the 

8 multimedia document occurring between t5 and tg, the representation of the multimedia 

9 information stored in the multimedia document occurring between 1 5 and displayed in the 

1 0 second area comprising a representation of information of the first type occurring between t5 and 

1 1 t5 and a representation of information of the second type occurring between t5 and tg. 

1 21. (Currently amended) The method of claim 19 fiirther comprising: 

2 changing [[the]] position of the second lens in response to user input such that the 

3 second lens visually emphasizes a portion of the second area of the GUI comprising a 

4 representation of the multimedia information stored in the multimedia document occurring 

5 between a fifth time (t5) and a sixth time (tg), where (tj < t5 < tg < t2), (tf ^ t3), and (tg ^ 

6 and 

7 responsive to the change in the position of the second lens, automatically 

8 displaying, in the third area of the GUI, the representation of the multimedia information stored 

9 in the multimedia document occurring between t5 and t^, the representation of the multimedia 

1 0 information stored in the multimedia document occurring between t f and t ^ displayed in the 

1 1 third area comprising a representation of information of the first type occurring between t5 and 

12 tg and a representation of information of the second type occurring between t5 and tg. 



1 22. (Currently amended) The method of claim 19 further comprising: 

2 displaying , in the first area of the GUI, a third lens positionable over a plurality of 

3 positions within the first area of the GUI, the third lens visually emphasizing a portion of the first 

4 area of the GUI comprising a representation of tiie multimedia information stored in the 

5 multimedia document occurring between t3 and t4 

1 23. (Currently amended) The method of claim 22 fiuther comprising: 
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2 changing [[the]] position of the second lens in response to user input such that the 

3 second lens visually emphasizes a portion of the second area of the GUI comprising a 

4 representation of tiie multimedia information stored in the multimedia document occurring 

5 between a fifth time (t5) and a sixth time (tg), where (tj < t5 < tg < t2), (tf ^ t3), and (tg ^ t4); 

6 and 

7 responsive to the change in the position of the second lens, automatically 

8 changing [[the]] position of the third lens such that the third lens visually emphasizes a portion of 

9 the first area of the GUI comprising a representation of tiie multimedia information stored in the 
1 0 multimedia document occurring between t5 and % 

1 24. (Currently amended) The method of claim 18 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises audio information; 

4 the representation of the information of the first type occurring between tg and t g 

5 comprises one or more video keyframes extracted from the video information; and 

6 the representation of information of the second type occurring between tg and tg 

7 comprises text information obtained from transcribing the audio information. 

1 25. (Currently amended) The method of claim 18 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises closed-caption (CC) text 

4 information; 

5 the representation of the information of the first type occurring between tg andtg 

6 comprises one or more video keyframes extracted from the video information; and 

7 the representation of the information of the second type occurring between tg and 

8 tg_comprises text information included in the CC text information. 

1 26. (Currently amended) The method of claim 18 further comprising: 

2 receiving information indicating a first topic; and 
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3 analyzing the multimedia information stored in the multimedia document to 

4 identify one or more locations in the multimedia information that are relevant to the first topic; 

5 wherein displayin g, in the first area of the GUI, the representation of the 

6 multimedia information stored [[by]] in the multimedia document occurring between tg and tg m 

7 the first area of the GUI comprises highlighting the one or more locations in the multimedia 

8 information displayed in the first area of the GUI that are relevant to the first topic ; and 

9 wherein displayin g, in the second area of the GUI, the representation of the 

1 0 multimedia information stored in the multimedia document occurring between tj and t2 intho 

1 1 second area of the GUI comprises highlighting the one or more locations in the multimedia 

12 information displayed in the second area of the GUI that are relevant to the first topic and that 

13 occur between times 1 1 and t2 . 

1 27. (Original) The method of claim 18 further comprising: 

2 receiving input indicating selection of a portion of the multimedia information 

3 occurring between a selection start time and a selection end time; and 

4 performing a first operation on the portion of the multimedia information 

5 occurring between the selection start time and the selection end time. 

1 28. (Currently amended) A method of displaying multimedia information 

2 stored in a multimedia document on a display, th e multimedia information comprising vid e o 

3 information and information of a first t3^ e , the method comprising: 

4 displaying a graphical user interface (GUI) on the display; 

5 displaying, in a first section of a first area of the GUI, a first set of one or more 

6 video keyframes extracted from [[the]] video information stored in the multimedia document 

7 occurring between a start time (tg) and an end time (tg) associated with the multimedia 

8 document, where (tg > t^); 

9 displaying, in a second section of the first area of the GUI, text information 

1 0 corresponding to [[the]] information of [[the]] a first type stored in the multimedia document 

1 1 occurring between tg and tg; 
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12 displaying a first lens movoablo in response to user input over roprosontations of 

13 multimedia information displayed in the GUI positionable over a plurality of portions of the first 

14 area of the GUI , the first lens emphasizing a portion of the first section of the first area occurring 

15 between a first time (tj) and a second time (t2) and a portion of the second section of the first 

16 area occurring between tj and t2, the emphasized portion of the first section of the first area 

1 7 comprising a second set of one or more video keyfi-ames extracted from the video information 

1 8 occurring between tj and t2, the emphasized portion of the second section of the first area 

19 comprising text information corresponding to information of the first type occurring between t\ 

20 and t2, wherein the second set of one or more keyframes [[is]] comprises a subset of the first set 

2 1 of one or more keyframes and (tg < tj < t2 < tg); 

22 displaying , in a first section of a second area of the GUI, the second set of one or 

23 more keyframes based on the first lens emphasizing the portion of the first section of the first 

24 area in a first s e ction of a second area of the GUI ; and 

25 displayin g, in a second section of the second area of the GUI, text information 

26 corresponding to the information of the first type occurring between tj and t2 based on the first 

27 lens emphasizing the portion of the second section of the first area in a s e cond s e ction of th e 

28 second area of the GUI . 

1 29. (Currently amended) The method of claim 28 further comprising: 

2 displaying a second lens movoablo in response to user input over roprosontations 

3 of multimedia information displayed in the GUI positionable over a plurality of portions of the 

4 second area of the GUI , the second lens emphasizing a portion of the first section of the second 

5 area and a portion of the second section of the second area, the emphasized portion of the first 

6 section of the second area comprising a third set of one or more video keyframes extracted from 

7 the video information occurring between a third time (t3) and a fourth time (t4), the emphasized 

8 portion of the second section of the second area comprising text information corresponding to 

9 information of the first type occurring between t3 and t^, wherein the third set of one or more 
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10 video keyframes [[is]] comprises a subset of the second set of one or more video keyframes and 

11 (ti<t3<t4<t2); 

1 2 displaying , in a First section of a third area of the GUI, a keyframe from the third 

1 3 set of one or more keyframes based on the second lens emphasizing the portion of the first 

14 section of the second area in a first section of a third area of tho GUI : and 

1 5 displayin g, in a second section of the third area of the GUI, text information 

1 6 corresponding to the information of the first type occurring between t3 and t4 based on the 

17 second lens emphasizing the portion of the second section of the second area in a second section 

18 of tho third area of tho GUI . 

1 30. (Currently amended) The method of claim 28 further comprising: 

2 displaying a second lens positionable over a plurality of portions within the 

3 second area of the GUI, the second lens emphasizing a portion of the first section of the second 

4 area and a portion of the second section of the second area, the emphasized portion of the first 

5 section of the second area comprising a third set of one or more video keyframes exfracted from 

6 the video information occurring between a third time (t3) and a fourth time (t4), the emphasized 

7 portion of the second section of the second area comprising text information corresponding to 

8 information of the first type occurring between t3 and 14, wherein the third set of one or more 

9 video keyframes [[is]] comprises a subset of the second set of one or more video keyframes and 
10 (ti<t3<t4<t2); 



1 1 outputting video information starting from 13 or from t4 or from a time between t3 

12 and 14 in a first section of a third area of the GUI; and 

1 3 displaying text information corresponding to the information of the first type 

14 occurring between 13 and t4 in a second section of the third area of the GUI based on the first 

15 lens emphasizing the portion of the second section of the second area . 

1 31. (Currently amended) The method of claim 28 wherein the information of 

2 the first type between t ^ and t o [[is]] comprises audio information, and the text information 
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3 corresponding to the information of the first type occurring between t ^ and t ? is obtained fi-om 

4 transcribing the audio information. 

1 32. (Currently amended) The method of claim 28 wherein the information of 

2 the first type between t ^ and t ? [[is]] comprises closed-caption (CC) text information, and the 

3 text information corresponding to the information of the first type between t ^ and t ? is extracted 

4 fi-om the CC text information. 

1 33. (Currently amended) The method of claim 28 wherein the multimedia 

2 information stored [[by]] m the multimedia document fiirther comprises slides information, the 

3 method comprising: 

4 displaying, in a third section of the first area of the GUI, a first set of one or more 

5 slides extracted from the slides information occurring between tg and tg, wherein the first lens 

6 emphasizes a portion of the third section of the first area comprising a second set of one or more 

7 slides extracted from the slides information occurring between tj and t2, the second set of one or 

8 more slides [[is]] comprising a subset of the first set of one or more slides; and 

9 displaying the second set of one or more slides in a third section of the second 

1 0 area of the GUI based on the first lens emphasizing the portion of the third section of the first 

11 area . 

1 34. (Currently amended) The method of claim 33 fiirther comprising: 

2 displaying a second lens positionable over a plurality of portions of the second 



3 area, the second lens emphasizing a portion of the first section of the second area, a portion of 

4 the second section of the second area, and a portion of the third section of the second area, the 

5 emphasized portion of the first section of the second area comprising a third set of one or more 

6 video keyframes extracted from the video information occurring between a third time (t3) and a 

7 fourth time (t4), the emphasized portion of the second section of the second area comprising text 

8 information corresponding to information of the first type occurring between t3 and t^, the 

9 emphasized portion of the third section of the second area comprising a third set of one or more 
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1 0 slides extracted from the slides information occurring between t3 and 14, wherein the third set of 

1 1 one or more video keyframes [[is]] comprises a subset of the second set of one or more video 

12 keyframes, the third set of one or more slides [[is]] comprising a subset of the second set of one 

1 3 or more slides, and (tj < t3 < t4 < t2); 

14 displayin g, in a first section of a third area of the GUI, at least one keyframe from 

1 5 the third set of one or more video keyframes based on the second lens emphasizing the portion of 

16 the first section of the second area in a first section of a third area of the GUI : 

1 7 displavin g. in a second section of the third area of the GUI, the text information 

1 8 corresponding to the information of the first type occurring between 13 and 14 based on the 

19 second lens emphasizing the portion of the second section of the second area in a second aootion 

20 of the third area of the GUI ; and 

2 1 displavin g. in a third section of the third area of the GUI, at least one slide from 

22 the third set of one or more shdes based on the second lens emphasizing the portion of the third 

23 section of the second area in a third section of the third area of the GUI . 

1 35. (Currently amended) The method of claim 28 wherein the multimedia 

2 information stored [[by]] in the multimedia document fiirther comprises whiteboard images 

3 information, the method comprising: 

4 displaying, in a third section of the first area of the GUI, a first set of one or more 

5 whiteboard images extracted from the whiteboard images information occurring between tg and 

6 tg, wherein the first lens emphasizes a portion of the third section of the first area comprising a 

7 second set of one or more whiteboard images extracted from the whiteboard images information 

8 occurring between t\ and t2, the second set of one or more whiteboard images [[is]] comprising a 

9 subset of the first set of one or more whiteboard images; and 

1 0 displayin g, in a third section of the second area of the GUI, the second set of one 

11 or more whiteboard images based on the first lens emphasizing the portion of the third section of 

12 the first area in a third s e ction of th e s e cond ar e a of th e GUI . 

1 36. (Currently amended) The method of claim 35 fiirther comprising: 
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2 displaying a second lens positionable over a plurality of portions of the second 

3 area of the GUI, the second lens emphasizing a portion of the first section of the second area, a 

4 portion of the second section of the second area, and a portion of the third section of the second 

5 area, the emphasized portion of the first section of the second area comprising a third set of one 

6 or more video keyframes extracted from the video information occurring between a third time 

7 (t3) and a fourth time (t4), the emphasized portion of the second section of the second area 

8 comprising text information corresponding to information of the first type occurring between t3 

9 and t^, the emphasized portion of the third section of the second area comprising a third set of 

10 one or more whiteboard images extracted from the whiteboard images information occurring 

1 1 between t3 and t4, wherein the third set of one or more video keyframes [[is]] comprises a subset 

12 of the second set of one or more video keyframes, the third set of one or more whiteboard images 

1 3 [[is]] comprising a subset of the second set of one or more whiteboard images, and (tj < t3 < t4 < 

14 t2); 



1 5 displayin g, in a first section of the third area of the GUI, at least one keyframe 

1 6 from the third set of one or more video keyframes based on the second lens emphasizing the 

17 portion of the first section of the second area in a first section of a third area of the GUI : 

18 displaying , in a second section of the third area of the GUI, the text information 

19 corresponding to the information of the first type occurring between 13 and 14 based on the 

20 second lens emphasizing the portion of the second section of the second area in a second section 

21 of the third area of the GUI ; and 

22 displavin g. in a third section of the third area of the GUI, a whiteboard image[[s]] 

23 from the third set of one or more whiteboard images based on the second lens emphasizing the 

24 portion of the third section of the second area in a third section of the third area of the GUI . 

1 37. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document on a display[[,]] the multimedia information comprising 

3 information of a plurality of types including information of a first typo and information of a 

4 second typo, the system comprising: 
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5 means for displaying a graphical user interface (GUI) on the display; 

6 moans for displaying, in a first area of the GUI, a roprosontation of the multimedia 

7 information stored by the multimedia docum e nt, th e display e d r e pr e s e ntation of th e multim e dia 

8 information comprising a representation of information of th e first typ e and a r e pr e s e ntation of 

9 information of the second type; 

10 means for displaying, in a first area of the GUI, a first visual representation of the 

11 multimedia information stored in the multimedia document, the first visual representation 

12 including a first representation of information of a first type stored in the multimedia document 

13 and a first representation of information of a second type stored in the multimedia document; 

1 4 means for displayin g, in the first area of the GUI, a first lens moveable in 

15 response to user input over representations of multimedia information displayed in the GUI 

16 positionable over a pluralitv of portions of the first visual representation within the first area of 

1 7 the GUI , the first lens covering a first portion of the first visual representation within the first 

1 8 area; and 

19 m e ans for displaying, in a s e cond ar e a of th e GUI, a r e pr e s e ntation of multim e dia 



20 information display e d in th e first portion of th e first ar e a, th e r e pr e s e ntation of multim e dia 

21 information displayed in the second area comprising a portion of the representation of 

22 information of the first type covorod by the First lens and a portion of the representation of 

23 information of the second typo covorod by the first Ions 

24 means for displaying, in a second area of the GUI, a second visual representation 

25 of the multimedia information stored in the multimedia document based on the first lens covering 

26 the first portion of the first visual representation within the first area, the second visual 

27 representation including a second representation of the information of the first type stored in the 

28 multimedia document and a second representation of the information of the second type stored in 

29 the multimedia document . 

1 38. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document on a display, the multimedia information comprising 

3 information of a first type and information of a second typ e , the system comprising: 
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4 



5 



means for displaying a graphical user interface (GUI) on the display; 

means for displaying, in a first area of the GUI, a representation of the multimedia 



6 information stored [[by]] m the multimedia document occurring between a start time (tg) and an 

7 end time (tg) associated with the multimedia document, the displayed representation of the 

8 multimedia information comprising a representation of information of [[the]] a first type stored 

9 in the multimedia document occurring between tg and tg and a representation of information of 

1 0 [[the]] a second type stored in the multimedia document occurring between tg and tg, where (tg > 



13 respons e to user input ov e r r e pr e s e ntations of multim e dia information display e d in th e GUI 

14 positionable over a plurality of positions within the first area of the GUI , the first lens visually 

15 emphasizing a portion of the first area of the GUI covered by the first lens , the portion of the first 

16 area visually emphasized by the first lens comprising a representation of multimedia information 

1 7 stored in the multimedia document occurring between a first time (tj) and a second time (t2), 

1 8 where (tg < ti < t2 < tg); and 

1 9 means for displaying, in a second area of the GUI, the representation of the 

20 multimedia information stored in the multimedia document occurring between t \ and t2 based on 

21 the first lens visually emphasizing the representation of the multimedia information stored in the 

22 multimedia document occurring between t ^ and t o. the representation of tiie multimedia 

23 information stored in the multimedia document displayed in the second area comprising a 

24 representation of information of the first type occurring between tj and t2 and a representation of 

25 information of the second type occurring between tj and t2. 

1 39. (Currently amended) A system [[for]] of displaying multimedia 

2 information stored in a multimedia document on a display, the multimedia information 

3 comprising vid e o information and information of a first typ e , the system comprising: 

4 means for displaying a graphical user interface (GUI) on the display; 



11 ts); 



12 



means for displayin g, in the first area of the GUI, a first lens mov e abl e u 



Page 20 of 69 



Appl. No. 10/081,129 PATENT 



Amdt. dated September 29, 2008 
Reply to Office Action of April 28, 2008 

5 means for displaying, in a first section of a first area of the GUI, a first set of one 

6 or more video keyframes extracted from [[the]] video information stored in the multimedia 

7 document occurring between a start time (tg) and an end time (tg) associated with the multimedia 

8 document, where (tg > t^); 

9 means for displaying, in a second section of the first area of the GUI, text 

1 0 information corresponding to [[the]] information of [[the]] a first type stored in the multimedia 

1 1 document occurring between and tg; 

12 means for displaying a first lens moveable in response to user input over 



13 representations of multimedia information displayed in the GUI positionable over a plurality of 

14 positions within the first area of the GUI, the first lens emphasizing a portion of the first section 

1 5 of the first area occurring between a first time (tj) and a second time (t2) and a portion of the 

16 second section of the first area occurring between tj and t2, the emphasized portion of the first 

17 section of the first area comprising a second set of one or more video keyframes extracted from 

1 8 the video information occurring between tj and t2, the emphasized portion of the second section 

1 9 of the first area comprising text information corresponding to information of the first type 

20 occurring between t j and t2, wherein the second set of one or more keyframes [[is]] comprises a 

21 subset of the first set of one or more keyframes and (t^ < < t2 < tg); 

22 means for displayin g, in a first section of a second area of the GUI, the second set 

23 of one or more keyframes based on the first lens emphasizing the portion of the first section of 

24 the first area in a first section of a second area of the GUI ; and 

25 means for displayin g, in a second section of the second area of the GUI, the text 

26 information corresponding to the information of the first type occurring between t j and t2 based 

27 on the first lens emphasizing the portion of the second section of the first area in a second section 

28 of the second area of the GUI . 

1 40. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multimedia information comprising information of a plurality of 
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4 typos including information of a first typo and information of a second typo, the computer 

5 program product comprising: 

6 code for displaying a graphical user interface (GUI) on the display; 

7 code for displaying, in a first area of the GUI, a representation of the multimedia 

8 information stored by the multimedia document, the displayed representation of the multimedia 

9 information comprising a representation of information of tho first typo and a roprosontation of 

10 information of the second type; 

11 code for displaying, in a first area of the GUI, a first visual representation of the 

12 multimedia information stored in the multimedia document, the first visual representation 

13 including a first representation of information of a first type stored in the multimedia document 

14 and a first representation of information of a second type stored in the multimedia document; 

15 code for displaying a first lens moveable in response to user input over 

16 representations of multimedia information displayed in the GUI positionable over a plurality of 

17 portions of the first visual representation displayed within the first area of the GUI, the first lens 

1 8 covering a first portion of the first visual representation within the first area; and 

19 cod e for displaying, in a s e cond ar e a of th e GUI, a r e pr e s e ntation of multim e dia 

20 information displayed in tho first portion of the first area, tho roprosontation of multimedia 

21 information displayed in the second area comprising a portion of tho roprosontation of 

22 information of the first typo covered by the first ions and a portion of tho roprosontation of 

23 information of the second typ e cov e r e d by th e first lens 

24 code for displaying, in a second area of the GUI, a second visual representation of 

25 the multimedia information stored in the multimedia document based on the first lens covering 

26 the first portion of the first visual representation within the first area, the second visual 

27 representation including a second representation of the information of the first type stored in the 

28 multimedia document and a second representation of the information of the second type stored in 

29 the multimedia document . 
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1 41 . (Currently amended) The computer program product of claim 40 wherein 

2 the code for displaying the first visual representation of the multimedia information stored [[by]] 

3 in the multimedia document in the first area of the GUI comprises: 

4 code for displaying a first thumbnail image in the first area of the GUI, the first 

5 thumbnail image comprising the first representation of tiie information of the first type; and 

6 code for displaying a second thumbnail image in the first area of the GUI, the 

7 second thumbnail image comprising the first representation of tiie information of the second 

8 type. 

1 42. (Currently amended) The computer program product of claim 40 wherein 

2 the code for displaying, in the second area of the GUI, tho roprosontation of multimedia 

3 information display e d in the first portion of the first area displaying the second visual 

4 represnation of the multimedia information stored in the multimedia document comprises: 

5 code for displayin g, in a first sub-area of the second area of the GUI, the second 

6 representation of the information of the first type as a portion of the first representation of tiie 

7 information of the first type covered by the first lens in a first pan e l in th e s e cond ar e a of th e 

8 GUI; and 

9 code for displaying , in a second sub-area of the second area of the GUI, the 

1 0 second representation of the information of the second type as a portion of the first 

1 1 representation of tiie information of the second type covered by the first lens in a second panel in 

12 tho second area of the GUI . 

1 43. (Cturently amended) The computer program product of claim 40 wherein 

2 the code for displaying, in tho second area of tho GUI, tho roprosontation of multimedia 

3 information displayed in tho first portion of tho first area displaying the second visual 

4 representation of the multimedia information stored in the multimedia document comprises: 

5 code for determining a first time and a second time associated with the first lens; 
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6 code for displaying, in the second area of the GUI, a representation of information 

7 of the first type occurring between the first time and the second time associated with the first lens 

8 as the second representation of the information of the first type; and 

9 code for displaying, in the second area of the GUI, a representation of information 

10 of the second type occurring between the first time and the second time associated with the first 

1 1 lens as the second representation of the information of the second type . 

1 44. (Currently amended) The computer program product of claim 40 fiirther 

2 comprising: 

3 code for receiving user input moving the first lens over the first visual 

4 representation within the first area to cover a second portion of the first visual representation 

5 within the first area; and 

6 code for responsive to the user input, automatically changing the information 



7 second visual representation displayed in the second area of the GUI such that the second visual 

8 representation of the multimedia information stored in the multimedia document displayed in the 

9 second area of the GUI corresponds to the second portion of the first visual representation of the 

1 0 multimedia information stored in the multimedia document covered by the first lens includ e d in 

11 the second portion of the first area . 

1 45. (Currently amended) The computer program product of claim 40 further 

2 comprising: 

3 code for displayin g, in the second area of the GUI, a second lens mov e abl e in 

4 r e sponse to us e r input ov e r r e pr e s e ntations of multim e dia information display e d in th e GUI 

5 positionable over a plurality of portions of the second visual representation displayed within the 

6 second area of the GUI , the second lens covering a first portion of the second visual 

7 representation within the second area; and 

8 code for displaying, in a third area of the GUI, a third visual representation of the 

9 multimedia information stored in the multimedia document based on the second lens covering 

10 the first portion of the second visual representation within the second area corresponding to the 

1 1 first portion of the second area , the third visual representation of multim e dia information 
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1 2 displayed in tho third aroa comprising a portion of tho third representation of tiie information of 

1 3 the first type covered by tho second Ions and a portion of tho third representation of tiie 

1 4 information of the second type cov e red by th e s e cond l e ns . 

1 46. (Currently amended) The computer program product of claim 45 wherein 

2 the code for displaying, in the third area of the GUI, the third visual representation of tiie 

3 multimedia information stored in the multimedia document corresponding to tho first portion of 

4 the second area comprises: 

5 code for determining a first time and a second time associated with the second 

6 lens; 

7 code for displaying, in the third area of the GUI, a representation of the 

8 information of the first type occurring between the first time and the second time associated with 

9 the second lens as the third representation of the information of the first type : and 

1 0 code for displaying, in the third area of the GUI, a representation of tiie 

1 1 information of the second type occurring between the first time and the second time associated 

12 with the second lens as the third representation of the information of the second type . 

1 47. (Currently amended) The computer program product of claim 45 wherein: 

2 the code for displaying the first visual representation of the multimedia 

3 information stored [[by]] m the multimedia document in the first area of the GUI comprises: 

4 code for displaying a first thumbnail image in the first area of the GUI, the 

5 first thumbnail image comprising the first representation of tiie information of the first type; and 

6 code for displaying a second thumbnail image in the first area of the GUI, 

7 the second thumbnail image comprising the first representation of tiie information of the second 

8 type; 

9 the code for displaying the second visual representation of the multimedia 

1 0 information stored in the multimedia document displayed in the first portion of the first area in 

1 1 the second area of the GUI comprises: 
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12 code for displayin g, in a first sub-area of the second area of the GUI, the 

1 3 portion of the first representation of tiie information of the first type covered by the first lens in-a 

14 first panel in the second area of the GUI ; and 

15 code for displayin g, in a second sub-area of the second area of the GUI, 

16 the portion of the first representation of the information of the second type covered by the first 

17 lens in a second panel in the second area of the GUI ; and 

1 8 the code for displaying the third visual representation of the multimedia 

1 9 information stored in the multimedia document corr e sponding to th e first portion of th e s e cond 

20 a^ea in the third area of the GUI comprises: 

2 1 code for displayin g, in a first sub-area of the third area of the GUI, the 

22 portion of the second representation of the information of the first type covered by the second 

23 lens as the third representation of the information of the first type corresponding to the first 

24 portion of the second area of the GUI in a first sub - aroa of the third area of the GUI ; and 

25 code for displayin g, in a second sub-area of the third area of the GUI, the 

26 portion of the second representation of tiie information of the second type covered by the second 

27 lens as the third representation of the information of the second type corr e sponding to th e first 

28 portion of the second area of the GUI in a second sub - aroa of the third area of the GUI . 

1 48. (Currently amended) The computer program product of claim 45 fiirther 

2 comprising: 

3 code for receiving [[a]] user input moving the second lens over the second visual 

4 representation displayed within the second area to cover a second portion of the second visual 

5 representation within the second area; and 

6 responsive to the user input, code for automatically changing the information third 

7 visual representation displayed in the third area of the GUI such that the third visual 

8 representation of tiie multimedia information stored in the multimedia document displayed in the 

9 third area of the GUI corresponds to the second portion of the second visual representation of the 

1 0 multimedia information stored in the multimedia document covered by the second lens included 

11 in th e s e cond portion of the s e cond ar e a . 
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49. (Currently amended) The computer program product of claim 45 further 



2 



comprising: 



3 



code for receiving [[a]] user input moving the first lens er the first visual 



4 representation displaved within the first area to cover a second portion of the first visual 

5 representation within the first area; and 




8 second area of the GUI such that the second visual representation of the multimedia information 

9 stored in the multimedia document displayed in the second area of the GUI corresponds to the 

1 0 second portion of the first visual representation of the multimedia information stored in the 

11 multimedia document covered by the first lens includ e d in th e s e cond portion of th e first ar e a ; 

12 and 

1 3 changing the informati en third visual representation displayed in the third 

14 area of the GUI such that the third visual representation of the multimedia information stored in 

1 5 the multimedia document displayed in the third area of the GUI corresponds to the second visual 

1 6 representation of the multimedia information stored bv the multimedia document within includ e d 

1 7 in the second portion of the second area. 

1 50. (Currently amended) The computer program product of claim 45 further 

2 comprising: 

3 code for displaying a sub-lens covering a portion of the first visual representation 

4 displaved within the first area of the GUI corresponding to the first portion of the second visual 

5 representation within the second area of the GUI covered by the second lens. 

1 51. (Currently amended) The computer program product of claim 50 fiirther 

2 comprising: 

3 code for receiving [[a]] user input moving the second lens over the second visual 

4 representation displayed within the second area to cover a second portion of the second visual 

5 representation within the second area; and 




6 



responsive to the user input, code for automatically: 

changing the information second visual representation displayed in the 



7 
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6 responsive to the user input, code for automatically changing [[a]] position of the 

7 sub-lens to cover a portion of the first visual representation displayed within the first area of the 

8 GUI corresponding to the second visual representation within the second area covered by the 

9 second lens . 

1 52. (Currently amended) The computer program product of claim 40 wherein: 

2 the information of the first type corresponds to video information; and 

3 the first representation of the information of the first type comprises one or more 

4 video keyframes extracted from the video information. 

1 53. (Currently amended) The computer program product of claim 52 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the first representation of information of the second type comprises text 

4 information obtained from transcribing the audio information. 

1 54. (Currently amended) The computer program product of claim 52 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the first representation of information of the second type comprises text 

5 information included in the CC text information. 

1 55. (Currently amended) The computer program product of claim 40 further 

2 comprising: 

3 code for receiving information indicating a user-specified concept of interest; and 

4 code for analyzing the multimedia information stored in the multimedia document 

5 to identify one or more locations in the multimedia information that are relevant to the user- 

6 specified concept of interest; 

7 wherein the code for displayin g, in the first area of the GUI, the first visual 

8 representation of tiie multimedia information stored in the multimedia document in th e first ar e a 
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9 of tho GUI comprises code for annotating the one or more locations in the multimedia 

1 0 information that are relevant to the user-specified concept of interest; and 

1 1 wherein the code for displaying, in the second area of the GUI, [[a]] the second 

12 visual representation of the multimedia information stored in the multimedia document displayed 

13 in tho first portion of the first area comprises code for annotating the one or more locations in the 

14 multimedia information that are relevant to the user-specified concept of interest and that are 

1 5 located in the first portion of the first visual representation covered by the first lens within the 

16 first area. 

1 56. (Original) The computer program product of claim 40 further comprising: 

2 code for receiving input indicating selection of a portion of the multimedia 

3 information occurring between a first time and a second time; and 

4 code for performing a first operation on the portion of the multimedia information 

5 occurring between a first time and a second time. 

1 57. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multim e dia information comprising information of a first typ e and 

4 information of a second typo, the computer program product comprising: 

5 code for displaying a graphical user interface (GUI) on the display; 

6 code for displaying, in a first area of the GUI, a representation of the multimedia 



7 information stored [[by]] m the multimedia document occurring between a start time (tg) and an 

8 end time (tg) associated with the multimedia document, the displayed representation of the 

9 multimedia information stored in the multimedia document occurring between tg and tg 

1 0 comprising a representation of information of [[the]] a first type stored in the multimedia 

1 1 document occurring between tg and tg and a representation of information of [[the]] a second 

12 type stored in the multimedia document occurring between tg and tg, where (tg > tg); 

13 code for displayin g, in the fist area of the GUI, a first lens moveable in response 

14 to us e r input ov e r r e pr e sentations of multim e dia information display e d in the GUI positionable 
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1 5 over a plurality of positions within the first area of the GUI, the first lens visually emphasizing a 

1 6 portion of the first area of the GUI covered by the first lens , the portion of the first area visually 

1 7 emphasized by the first lens comprising a representation of multimedia information stored in the 

1 8 multimedia document occurring between a first time (tj) and a second time (t2), where (tg < tj < 

19 t2<te);and 

20 code for displaying, in a second area of the GUI, the representation of the 

21 multimedia information stored in the multimedia document occurring between t\ and t2 based on 

22 the first lens visually emphasizing the portion of the first area , the representation of the 

23 multimedia information stored in the multimedia document occurring between t ^ and t ? 

24 displayed in the second area comprising a representation of information of the first type 

25 occurring between tj and t2 and a representation of information of the second type occurring 

26 between tj andt2. 

1 58. (Currently amended) The computer program product of claim 57 fiirther 

2 comprising: 

3 code for displayin g, in the second area of the GUI, a second lens movoablo in 

4 r e spons e to us e r input ov e r r e pr e s e ntations of multim e dia information display e d in th e GUI 

5 positionable over a plurality of positions within the second area of the GUI , the second lens 

6 visually emphasizing a portion of the second area of the GUI covered by the second lens, the 

7 portion of the second area visually emphasized by the second lens comprising a representation of 

8 the multimedia information stored in the multimedia document occurring between a third time 

9 (t3) and a fourth time (t4), where (tj < t3 < t4 < t2); and 

1 0 code for displaying, in a third area of the GUI, the representation of tiie 

1 1 multimedia information stored in the multimedia document occurring between 13 and 14 based on 

12 the second lens visually emphasizing the portion of the second area , the representation of the 

1 3 multimedia information stored in the multimedia document occurring between t 3 and t 4 

14 displayed in the third area comprising a representation of information of the first type occurring 
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15 between t3 and and a representation of information of the second type occurring between t3 

16 andt4. 



1 59. (Currently amended) The computer program product of claim 58 further 

2 comprising: 

3 code for changing [[the]] position of the first lens in response to user input such 

4 that the first lens visually emphasizes a portion of the first area of the GUI comprising a 

5 representation of tiie multimedia information stored in the multimedia document occurring 

6 between a fifth time (t5) and a sixth time (tg), where (tg < t5 < tg < tg), (t5 tj), and (tg t2); 

7 and 

8 responsive to the change in the position of the first lens, code for automatically 



9 displaying, in the second area of the GUI, the representation of the multimedia information 

1 0 stored in the multimedia document occurring between t5 and tg, the representation of tiie 

1 1 multimedia information stored in the multimedia document occurring between t 5 and t g 

12 displayed in the second area comprising a representation of information of the first type 

1 3 occurring between 15 and tg and a representation of information of the second type occurring 

14 between t5 and tg. 



1 60. (Ctirrently amended) The computer program product of claim 58 further 

2 comprising: 

3 code for changing [[the]] position of the second lens in response to user input 

4 such that the second lens visually emphasizes a portion of the second area of the GUI comprising 

5 a representation of the multimedia information stored in the multimedia document occurring 

6 between a fifth time (15) and a sixth time (tg), where (tj < t5 < tg < t2), (t5 ^ t^), and (tg ^ 14); 

7 and 

8 code for responsive to the change in the position of the second lens, automatically 

9 displaying, in the third area of the GUI, the representation of ftie multimedia information stored 
1 0 in the multimedia document occurring between t5 and tg, the representation of ftie multimedia 
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1 1 information stored in the multimedia document occurring between U and displayed in the 

12 third area comprising a representation of information of the first type occurring between 15 and 

13 t^ and a representation of information of the second type occurring between 15 and tg. 



1 61 . (Currently amended) The computer program product of claim 58 further 

2 comprising: 

3 code for displayin g, in the first area of the GUI, a third lens positionable over a 

4 plurality of positions within the first area of the GUI, the third lens visually emphasizing a 

5 portion of the first area of the GUI comprising a representation of tiie multimedia information 

6 stored in the multimedia document occurring between t3 and t4 

1 62. (Currently amended) The computer program product of claim 61 fiirther 

2 comprising: 

3 code for changing [[the]] position of the second lens in response to user input 

4 such that the second lens visually emphasizes a portion of the second area of the GUI comprising 

5 a representation of the multimedia information stored in the multimedia document occurring 

6 between a fifth time (t5) and a sixth time (tg), where (tj < t5 < tg < t2), (t5 ^ t3), and (t^ ^ t^); 

7 and 

8 code for responsive to the change in the position of the second lens, automatically 

9 changing the position of the third lens such that the third lens visually emphasizes a portion of 

1 0 the first area of the GUI comprising a representation of the multimedia information stored in the 

1 1 multimedia document occurring between t5 and tg 

1 63. (Currently amended) The computer program product of claim 57 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises audio information; 

4 the representation of the information of the first type occurring between tg and tg 

5 comprises one or more video keyframes extracted fi-om the video information; and 
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6 the representation of information of the second type occurring between tg and t g 

7 comprises text information obtained from franscribing the audio information. 

1 64. (Currently amended) The computer program product of claim 57 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises closed-caption (CC) text 

4 information; 

5 the representation of the information of the first type occurring between tg and t g 

6 comprises one or more video keyframes extracted from the video information; and 

7 the representation of the information of the second type occurring between tg and 

8 tg comprises text information included in the CC text information. 

1 65. (Currently amended) The computer program product of claim 57 fiirther 

2 comprising: 

3 code for receiving information indicating a first topic; and 

4 code for analyzing the multimedia information stored in the multimedia document 

5 to identify one or more locations in the multimedia information that are relevant to the first topic; 

6 wherein the code for displavin g. in the first area of the GUI, the representation of 

7 the multimedia information stored [[by]] in the multimedia document occurring between tg and 

8 tg in the first area of the GUI comprises code for highlighting the one or more locations in the 

9 multimedia information displayed in the first area of the GUI that are relevant to the first topic ; 

10 and 

1 1 wherein the code for displavin g. in the second are of the GUI, the representation 

12 of the multimedia information stored in the multimedia document occurring between 1 1 and t2 m 

13 the second area of the GUI comprises code for highlighting the one or more locations in the 

14 multimedia information displayed in the second area of the GUI that are relevant to the first topic 

1 5 and that occur between times tj and t2. 

1 66. (Original) The computer program product of claim 57 fiirther comprising: 
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2 code for receiving input indicating selection of a portion of the multimedia 

3 information occurring between a selection start time and a selection end time; and 

4 code for performing a first operation on the portion of the multimedia information 

5 occurring between the selection start time and the selection end time. 

1 67. (Currently amended) A computer program product stored on a computer- 

2 readable storage medium for displaying multimedia information stored in a multimedia 

3 document on a display, the multimedia information comprising video information and 

4 information of a First type, the computer program product comprising: 

5 code for displaying a graphical user interface (GUI) on the display; 

6 code for displaying, in a first section of a first area of the GUI, a first set of one or 

7 more video keyframes extracted from [[the]] video information stored in a multimedia document 

8 occurring between a start time (tg) and an end time (tg) associated with the multimedia 

9 document, where (tg > tg); 

10 code for displaying, in a second section of the first area of the GUI, text 

1 1 information corresponding to [[the]] information of [[the]] a first type stored in a multimedia 

1 2 document occurring between tg and tg; 

13 code for displaying a first lens moveable in response to user input over 



14 representations of multimedia information displayed in the GUI positionable over a plurality of 

15 portions of the first area of the GUI , the first lens emphasizing a portion of the first section of the 

1 6 first area occurring between a first time (tj) and a second time (t2) and a portion of the second 

1 7 section of the first area occurring between tj and t2, the emphasized portion of the first section of 

1 8 the first area comprising a second set of one or more video keyframes extracted from the video 

19 information occurring between tj and t2, the emphasized portion of the second section of the 

20 first area comprising text information corresponding to information of the first type occurring 

21 between tj and t2, wherein the second set of one or more keyframes [[is]] comprises a subset of 

22 the first set of one or more keyframes and (tg < tj < t2 < tg); 



Page 34 of 69 



Appl. No. 10/081,129 PATENT 
Amdt. dated September 29, 2008 
Reply to Office Action of April 28, 2008 

23 code for displayin g, in a first section of a second area of the GUI, the second set 

24 of one or more keyfi-ames based on the first lens emphasizing the portion of the first section of 

25 the first area in a first section of a second area of the GUI ; and 

26 code for displaying , in a second section of the second area of the GUI, the text 

27 information corresponding to the information of the first type occurring between tj and t2 based 

28 on the first lens emphasizing the portion of the second section of the first area in a s e cond s e ction 

29 of the second area of the GUI . 

1 68. (Currently amended) The computer program product of claim 67 further 

2 comprising: 

3 code for displaying a second lens mov e abl e in r e spons e to us e r input ov e r 

4 roprosontations of multimedia information displayed in the GUI positionable over a plurality of 

5 portions of the second area of the GUI , the second lens emphasizing a portion of the first section 

6 of the second area and a portion of the second section of the second area, the emphasized portion 

7 of the first section of the second area comprising a third set of one or more video keyframes 

8 extracted from the video information occurring between a third time (t3) and a fourth time (t4), 

9 the emphasized portion of the second section of the second area comprising text information 

10 corresponding to information of the first type occurring between 13 and 14, wherein the third set 

11 of one or more video keyframes [[is]] comprises a subset of the second set of one or more video 

12 keyframes and (tj < t3 < t4 < t2); 



1 3 code for displayin g, in a first section of a third area of the GUI, a keyframe from 

1 4 the third set of one or more keyframes based on the second lens emphasizing the portion of the 

15 first section of the section area in a first section of a third area of the GUI ; and 

16 code for displayin g, in a second section of the third area of the GUI, text 

1 7 information corresponding to the information of the first type occurring between t3 and t4 based 

18 on the second lens emphasizing the portion of the second section of the section area in a s e cond 

19 section of the third area of the GUI . 
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69. 



(Currently amended) The computer program product of claim 67 further 



2 



comprising: 



3 



code for displaying a second lens positionable over a plurality of portions of the 



4 second area of the GUI, the second lens emphasizing a portion of the first section of the second 

5 area and a portion of the second section of the second area, the emphasized portion of the first 

6 section of the second area comprising a third set of one or more video keyfi'ames extracted fi-om 

7 the video information occurring between a third time (13) and a fourth time (t^), the emphasized 

8 portion of the second section of the second area comprising text information corresponding to 

9 information of the first type occurring between t3 and t^, wherein the third set of one or more 

10 video keyframes [[is]] comprises a subset of the second set of one or more video keyfi'ames and 

11 (ti<t3<t4<t2); 

12 code for outputting video information starting from t3 or from t4 or from a time 

1 3 between t3 and in a first section of a third area of the GUI; and 

14 code for displaying text information corresponding to the information of the first 

15 type occurring between 13 and 14 in a second section of the third area of the GUI based on the 

16 first lens emphasizing the portion of the second section of the second area . 

1 70. (Currently amended) The computer program product of claim 67 wherein 

2 the information of the first type between t ^ and t o [[is]] comprises audio information, and the 

3 text information corresponding to the information of the first type between t ^ and t o is obtained 

4 from transcribing the audio information. 

1 71. (Currently amended) The computer program product of claim 67 wherein 

2 the information of the first type between t ^ and t 2 [[is]] comprises closed-caption (CC) text 

3 information, and the text information corresponding to the information of the first type between 

4 ti andt2 [[is]] is exfracted from the CC text information. 
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1 72. (Currently amended) The computer program product of claim 67 wherein 

2 the multimedia information stored [[by]] in the multimedia document further comprises slides 

3 information, the computer program product further comprising: 

4 code for displaying, in a third section of the first area of the GUI, a first set of one 

5 or more slides extracted from the slides information occurring between t^ and tg, wherein the 

6 first lens emphasizes a portion of the third section of the first area comprising a second set of one 

7 or more slides extracted from the slides information occurring between t\ and t2, the second set 

8 of one or more slides [[is]] comprising a subset of the first set of one or more slides; and 

9 code for displaying the second set of one or more slides in a third section of the 

1 0 second area of the GUI based on the first lens emphasizing the portion of the third section of the 

11 first area . 

1 73. (Currently amended) The computer program product of claim 72 further 

2 comprising: 

3 code for displaying a second lens positionable over a plurality of portions of the 



4 second area, the second lens emphasizing a portion of the first section of the second area, a 

5 portion of the second section of the second area, and a portion of the third section of the second 

6 area, the emphasized portion of the first section of the second area comprising a third set of one 

7 or more video keyframes exfracted from the video information occurring between a third time 

8 (t3) and a fourth time (t4), the emphasized portion of the second section of the second area 

9 comprising text information corresponding to information of the first type occurring between t3 

1 0 and t4, the emphasized portion of the third section of the second area comprising a third set of 

1 1 one or more slides exfracted from the slides information occurring between t3 and t^, wherein 

12 the third set of one or more video keyframes [[is]] comprises a subset of the second set of one or 

13 more video keyframes, the third set of one or more slides [[is]] comprising a subset of the second 

14 set of one or more slides, and (tj < t3 < t4 < t2); 
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1 5 code for displayin g, in a first section of a third area of the GUI, at least one 

1 6 keyfi-ame fi-om the third set of one or more video keyframes based on the second lens 

17 emphasizing the portion of the first section of the second area in a first s e ction of a third ar e a of 

19 code for displaying , in a second section of a third area of the GUI, text 

20 information corresponding to the information of the first type occurring between t3 and based 

21 on the second lens emphasizing the portion of the second section of the second area in a second 

22 section of the third area of the GUI ; and 

23 displayin g, in a third section of the third area of tlic GUI, at least one slide from 

24 the third set of one or more shdes based on the second lens cmphasizina the portion of the third 

25 section of the second area in a third section of the third area of the GUI . 

1 74. (Currently amended) The computer program product of claim 67 wherein 

2 the multimedia information stored [[by]] in the multimedia document further comprises 

3 whiteboard images information, the computer program product further comprising: 

4 code for displaying, in a third section of the first area of the GUI, a first set of one 



5 or more whiteboard images exfracted from the whiteboard images information occurring 

6 between t^ and tg, wherein the first lens emphasizes a portion of the third section of the first area 

7 comprising a second set of one or more whiteboard images exfracted from the whiteboard images 

8 information occurring between t\ and t2, the second set of one or more whiteboard images [[is]] 

9 comprising a subset of the first set of one or more whiteboard images; and 



1 0 code for displayin g, in a third section of the second area of the GUI, the second 

1 1 set of one or more whiteboard images based on the first lens emphasizing the portion of the third 

12 section of the first area in a third s e ction of th e s e cond ar e a of th e GUI . 

1 75. (Currently amended) The computer program product of claim 74 fiirther 

2 comprising: 

3 code for displaying a second lens positionable over a plurality of portions of the 

4 second area of the GUI, the second lens emphasizing a portion of the first section of the second 
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5 area, a portion of the second section of the second area, and a portion of the third section of the 

6 second area, the emphasized portion of the first section of the second area comprising a third set 

7 of one or more video keyframes extracted from the video information occurring between a third 

8 time (13) and a fourth time (14), the emphasized portion of the second section of the second area 

9 comprising text information corresponding to information of the first type occurring between t3 

1 0 and t4, the emphasized portion of the third section of the second area comprising a third set of 

1 1 one or more whiteboard images extracted from the whiteboard images information occurring 

12 between t3 and t4, wherein the third set of one or more video keyframes [[is]] comprises a subset 

13 of the second set of one or more video keyframes, the third set of one or more whiteboard images 

14 [[is]] comprising a subset of the second set of one or more whiteboard images, and (tj < t3 < t4 < 

15 t2); 



16 code for displaying , in a first section of the third area of the GUI, at least one 

1 7 keyframe from the third set of one or more video keyframes based on the second lens 

18 emphasizing the portion of the first section of the second area in a first s e ction of a third ar e a of 

19 the GUI ' 

20 code for displayin g, in a second section of the third area of the GUI, the text 

21 information corresponding to the information of the first type occurring between 13 and 14 based 

22 on the second lens emphasizing the portion of the second section of the second area in a second 

23 Qoction of the third area of the GUI ; and 

24 code for displayin g, in a third section of the third area of the GUI, a whiteboard 

25 image[[s]] from the third set of one or more whiteboard images based on the second lens 

26 emphasizing the portion of the third section of the second area in a third section of the third aroa 

27 of the GUI . 

1 76. (Currently amended) A system for displaying multimedia information 

2 stored in a multimedia document, the multimedia information comprising information of a 

3 plurality of types including information of a first type and information of a second type, the 

4 system comprising: 
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5 a display; 

6 a processor; and 

7 a memory coupled to the processor, the memory configured to store a plurality of 

8 code modules for execution by the processor, the plurality of code modules comprising: 



9 a code module for displaying a graphical user interface (GUI) on the 

10 display; 

11 a code module for displaying, in a first ar e a of th e GUI, a r e pr e s e ntation of 

12 the multimedia information stor e d by th e multim e dia docum e nt, th e display e d r e pr e s e ntation of 

13 the multimedia information comprising a representation of information of the first typo and a 

14 roprosontation of information of the second iypo; 

15 a code module for displaying, in a first area of the GUI, a first visual 

16 representation of the multimedia information stored in the multimedia document, the first visual 

17 representation including a first representation of information of a first type stored in the 

18 multimedia document and a first representation of information of a second type stored in the 

19 multimedia document 

20 a code module for displayin g, in the first area of the GUI, a first lens 

21 moy cable in response to user input over representations of multimedia information displayed in 

22 the GUI positionabic over a plurality of portions of the first visual representation displayed 

23 yyithin the first area of the GUI , the first lens covering a first portion of the first visual 

24 representation within the first area; and 

25 a cod e modul e for displaying, in a s e cond ar e a of th e GUI, a 

26 roprosontation of multimedia information displayed in the first portion of the first area, the 

27 roprosontation of multimedia information displayed in the second area comprising a portion of 

28 th e r e pr e s e ntation of information of th e first t3^ e cov e r e d by th e first l e ns and a portion of th e 

29 r e pr e s e ntation of information of the second type covered by the first lens 

30 a code module for displaying, in a second area of the GUI, a second visual 

31 representation of the multimedia information stored in the multimedia document based on the 

32 first lens covering the first portion of the first visual representation yyithin the first area, the 

33 second visual representation including a second representation of the information of the first type 
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34 Stored in the multimedia document and a second representation of the information of the second 

35 type stored in the multimedia document . 

1 77. (Currently amended) The system of claim 76 wherein the code module for 

2 displaying the first visual representation of the multimedia information stored [[by]] in the 

3 multimedia document in the first area of the GUI comprises: 

4 a code module for displaying a first thumbnail image in the first area of the GUI, 

5 the first thumbnail image comprising the first representation of tiie information of the first type; 

6 and 

7 a code module for displaying a second thumbnail image in the first area of the 

8 GUI, the second thumbnail image comprising the first representation of tiie information of the 

9 second type. 

1 78. (Currently amended) The system of claim 76 wherein the code module for 

2 displaying, in the second area of the GUI, the roprosontation of multimedia information 

3 displayed in the first portion of the first area displaying the second visual representation of the 

4 multimedia information stored in the multimedia document comprises: 

5 a code module for . in a first sub-area of the second area of the GUI, the second 

6 representation of the information of the first type as a portion of the first representation of the 

7 information of the first type covered by the first lens in a first panel in the second area of the 

8 GUI; and 

9 a code module for displayin g, in a second sub-area of the second area of the GUI. 

1 0 the second representation of the information of the first type as a portion of the first 

1 1 representation of the information of the second type covered by the first lens in a second panel in 

12 the second area of the GUI . 

1 79. (Currently amended) The system of claim 76 wherein the code module for 

2 displaying, in the second area of the GUI. the representation of multimedia information 

3 displayed in the first portion of the first area displaying the second visual representation of the 

4 multimedia information stored in the multimedia document comprises: 
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5 a code module for determining a first time and a second time associated with the 

6 first lens; 

7 a code module for displaying, in the second area of the GUI, a representation of 

8 the information of the first type occurring between the first time and the second time associated 

9 with the first lens as the second representation of the information of the first type ; and 

10 a code module for displaying, in the second area of the GUI, a representation of 

1 1 the information of the second type occurring between the first time and the second time 

1 2 associated with the first lens as the second representation of the information of the second type . 

1 80. (Currently amended) The system of claim 76 wherein the plurality of 

2 code modules further comprises: 

3 a code module for receiving user input moving the first lens over the first visual 

4 representation displayed within the first area to cover a second portion of the first visual 

5 representation within the first area; and 

6 responsive to the user input, a code module for automatically changing the 

7 information second visual representation displayed in the second area of the GUI such that the 

8 second visual representation of tiie multimedia information stored in the multimedia document 

9 displayed in the second area of the GUI corresponds to the second portion of the first visual 

1 0 representation of tiie multimedia information stored in the multimedia document covered by the 

11 first lens included in the second portion of the first area . 

1 81. (Currently amended) The system of claim 76 wherein the plurality of 

2 code modules fiirther comprises: 

3 a code module for displaying , in the second area of the GUI, a second lens 

4 moveable in response to user input over roprosontations of multimedia information displayed in 

5 the GUI positionable over a plurality of portions of the second visual representation displayed 

6 within the second area of the GUI , the second lens covering a first portion of the second visual 

7 representation within the second area; and 

8 a code module for displaying, in a third area of the GUI, a third visual 

9 representation of tiie multimedia information stored in the multimedia document based on the 
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10 second lens covering the first portion of the second visual representation within the second area 

1 1 corresponding to the first portion of the second area , the third visual representation ef 

12 multimedia information displayed in the third area comprising including a portion of th e third 

1 3 representation of the information of the first type covered by the second lens and a portion of th e 

14 third representation of the information of the second type covered by the second Ions . 

1 82. (Currently amended) The system of claim 81 wherein the code module for 

2 displaying, in the third area of the GUI, the third visual representation of the multimedia 

3 information stored in the multimedia document corresponding to the first portion of the second 

4 ^ea comprises: 

5 a code module for determining a first time and a second time associated with the 

6 second lens; 

7 a code module for displaying, in the third area of the GUI, a representation of tiie 

8 information of the first type occurring between the first time and the second time associated with 

9 the second lens as the third representation of the information of the first type ; and 

10 a code module for displaying, in the third area of the GUI, a representation of the 

1 1 information of the second type occurring between the first time and the second time associated 

1 2 with the second lens as the third representation of the information of the second type . 

1 83. (Currently amended) The system of claim 81 wherein: 

2 the code module for displaying the first visual representation of the multimedia 

3 information stored [[by]] m the multimedia document in the first area of the GUI comprises: 

4 a code module for displaying a first thumbnail image in the first area of 

5 the GUI, the first thumbnail image comprising the first representation of tiie information of the 

6 first type; and 

7 a code module for displaying a second thumbnail image in the first area of 

8 the GUI, the second thumbnail image comprising the first representation of the information of 

9 the second type; 
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1 0 the code module for displaying the second visual representation of tiie multimedia 

1 1 information stored in the multimedia document displayed in the first portion of the first area in 

12 the second area of the GUI comprises: 

13 a code module for displaying , in a first sub-area of the second area of the 

14 GUI, the portion of the first representation of the information of the first type covered by the first 

15 lens as the second representation of the information of the first type in a first panel in the second 

16 area of the GUI ; and 

1 7 a code module for displayin g, in a second sub-area of the second area of 

1 8 the GUI, the portion of the first representation of tiie information of the second type covered by 

1 9 the first lens as the second representation of the information of the second type in a second panel 

20 in the second area of the GUI ; and 

2 1 the code module for displaying the third visual representation of the multimedia 

22 information stored in the multimedia document corresponding to the first portion of the second 

23 area in the third area of the GUI comprises: 

24 a code module for displayin g, in a first sub-area of the third area of the 

25 GUI, the portion of the second representation of tiie information of the first type covered bv the 

26 second lens as the third representation of the information of the first type corresponding to the 

27 first portion of the second area of the GUI in a first sub area of the third area of the GUI ; and 

28 a code module for displaying , in a second sub-area of the third area of the 

29 GUI, the portion of the second representation of the information of the second type covered by 

30 the second lens as the third representation of the information of the first tvpe corr e sponding to 

31 the first portion of the second area of the GUI in a second sub - area of the third area of the GUI . 

1 84. (Currently amended) The system of claim 81 wherein the plurality of 

2 code modules fiirther comprises: 

3 a code module for receiving [[a]] user input moving the second lens over the 

4 second visual representation displayed within the second area to cover a second portion of the 

5 second visual representation within the second area; and 
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6 responsive to the user input, a code module for automatically changing the 

7 information third visual representation displayed in the third area of the GUI such that the third 

8 visual representation of tiie multimedia information stored in the multimedia document displayed 

9 in the third area of the GUI corresponds to the second portion of the second visual representation 

10 of the multimedia information stored in the multimedia document covered by the second lens 

11 included in the second portion of the second area . 



1 85. (Currently amended) The system of claim 81 wherein the plurality of 

2 code modules further comprises: 

3 a code module for receiving [[a]] user input moving the first lens over the first 

4 visual representation displayed within the first area to cover a second portion of the first visual 

5 representation within first area; and 

6 responsive to the user input, a code module for automatically: 

7 changing the informati en second visual representation displayed in the 



8 second area of the GUI such that the second visual representation of the multimedia information 

9 stored in the multimedia document displayed in the second area of the GUI corresponds to the 

1 0 second portion of the first visual representation of tiie multimedia information stored in the 

11 multimedia document covered by the first lens included in the second portion of the first area : 

12 and 



1 3 changing the information third visual representation displayed in the third 

14 area of the GUI such that the third visual representation of the multimedia information stored in 

15 the multimedia document displayed in the third area of the GUI corresponds to the second visual 

1 6 representation of the multimedia information stored by the multimedia document within included 

17 in the second portion of the second area. 

1 86. (Currently amended) The system of claim 81 wherein the plurality of 

2 code modules further comprises: 

3 a code module for displaying a sub-lens covering a portion of the first visual 

4 representation displayed within the first area of the GUI corresponding to the first portion of the 

5 second visual representation within the second area of the GUI covered by the second lens. 
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1 87. (Currently amended) The system of claim 86 wherein the plurality of 

2 code modules further comprises: 

3 a code module for receiving [[a]] user input moving the second lens over the 

4 second visual representation displayed within the second area to cover a second portion of the 

5 second visual representation within the second area; and 

6 responsive to the user input, a code module for automatically changing [[a]] 

7 position of the sub-lens to cover a portion of the first visual representation displayed within the 

8 first area of the GUI corresponding to the second portion of the second visual representation 

9 within the second area covered by the second lens . 

1 88. (Currently amended) The system of claim 76 wherein: 

2 the information of the first type corresponds to video information; and 

3 the first representation of the information of the first type comprises one or more 

4 video keyframes extracted from the video information. 

1 89. (Ctjrrently amended) The system of claim 88 wherein: 

2 the information of the second type corresponds to audio information; and 

3 the first representation of tiie information of the second type comprises text 

4 information obtained fi-om transcribing the audio information. 

1 90. (Currently amended) The system of claim 88 wherein: 

2 the information of the second type corresponds to closed-caption (CC) text 

3 information; and 

4 the first representation of the information of the second type comprises text 

5 information included in the CC text information. 

1 91 . (Currently amended) The system of claim 76 wherein the plurality of 

2 code modules further comprises: 

3 a code module for receiving information indicating a user-specified concept of 

4 interest; and 
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5 a code module for analyzing the multimedia information stored in the multimedia 

6 document to identify one or more locations in the multimedia information that are relevant to the 

7 user-specified concept of interest; 

8 wherein the code module for displayin g, in the first area of the GUI, the first 

9 visual representation of the multimedia information stored in the multimedia document in the 

10 first area of the GUI comprises annotating the one or more locations in the multimedia 

1 1 information that are relevant to the user-specified concept of interest; and 

12 wherein the code module for displaying, in the second area of the GUI, [[a]] tiie 

1 3 second visual representation of tiie multimedia information stored in the multimedia document 

14 displayed in tho first portion of tho first area comprises annotating the one or more locations in 

15 the multimedia information that are relevant to the user-specified concept of interest and that are 

1 6 located in the first portion of the first visual representation covered by the first lens within the 

17 first area. 

1 92. (Original) The system of claim 76 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving input indicating selection of a portion of the 

4 multimedia information occurring between a first time and a second time; and 

5 a code module for performing a first operation on the portion of the multimedia 

6 information occurring between a first time and a second time. 

7 93. (Currently amended) A system for displaying multimedia information 

8 stored in a multimedia document, th e multim e dia information comprising information of a first 

9 typo and information of a second typo, the system comprising: 

10 a display; 

1 1 a processor; and 

12 a memory coupled to the processor, the memory configured to store a plurality of 

13 code modules for execution by the processor, the plurality of code modules comprising: 

14 a code module for displaying a graphical user interface (GUI) on the 

15 display; 
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16 a code module for displaying, in a first area of the GUI, a representation of 

1 7 the multimedia information stored [[by]] in the multimedia document occurring between a start 

1 8 time (tg) and an end time (tg) associated with the multimedia document, the display e d 

19 representation of the multimedia information stored in the multimedia document occurring 

20 between tg and t g comprising a representation of information of [[the]] a first type stored in the 

2 1 multimedia document occurring between tg and tg and a representation of information of [[the]] 

22 a second type stored in the multimedia document occurring between tg and tg, where (tg > tg); 

23 a code module for displaying , in the first area of the GUI, a first lens 

24 moveable in response to user input over rcprosentations of multimedia information displayed in 

25 the GUI positionable over a plurality of positions within the first area of the GUI , the first lens 

26 visually emphasizing a portion of the first area of the GUI covered by the first lens , the portion 

27 of the first area visually emphasized by the first lens comprising a representation of multimedia 

28 information stored in the multimedia document occurring between a first time (tj) and a second 

29 time (t2), where (tg < ti < t2 < tg); and 

30 a code module for displaying, in a second area of the GUI, the 

3 1 representation of the multimedia information stored in the multimedia document occurring 

32 between t] and t2 based on the first lens visually emphasizing the portion of the first area , the 

33 representation of the multimedia information stored in the multimedia document occurring 

34 between t i andt ? displayed in the second area comprising a representation of information of the 

35 first type occurring between t\ and t2 and a representation of information of the second type 

36 occurring between t j and t2. 



1 94. (Currently amended) The system of claim 93 wherein the plurality of 

2 code modules further comprises: 

3 a code module for displayin g, in the second area of the GUI, a second lens 

4 movoablo in response to user input over roprosontations of multimedia information displayed in 

5 th e GUI positionable over a plurality of positions within the second area of the GUI , the second 

6 lens visually emphasizing a portion of the second area of the GUI covered by the second lens, the 



Page 48 of 69 



Appl. No. 10/081,129 PATENT 
Amdt. dated September 29, 2008 
Reply to Office Action of April 28, 2008 

7 portion of the second area visually emphasized by the second lens comprising a representation of 

8 multimedia information stored in the multimedia document occurring between a third time (t3) 

9 and a fourth time (14), where (t^ < 13 < t4 < t2); and 

10 a code module for displaying, in a third area of the GUI, the representation of the 

1 1 multimedia information stored in the multimedia document occurring between t3 and t4 based on 

1 2 the second lens visually emphasizing the portion of the second area , the representation of the 

13 multimedia information stored in the multimedia document occurring between and 

14 displayed in the third area comprising a representation of information of the first type occurring 

1 5 between t3 and t4 and a representation of information of the second type occurring between t3 

16 andt4. 



1 95. (Currently amended) The system of claim 94 wherein the plurality of 

2 code modules further comprises: 

3 a code module for changing [[tlic]] position of the first lens in response to user 

4 input such that the first lens visually emphasizes a portion of the first area of the GUI comprising 

5 a representation of the multimedia information stored in the multimedia document occurring 

6 between a fifth time (t5) and a sixth time (tg), where (tg < t5 < tg < tg), (t5 ^ tj), and (tg ^ t2); 

7 and 

8 responsive to the change in the position of the first lens, a code module for 



9 automatically displaying, in the second area of the GUI, the representation of the multimedia 

1 0 information stored in the multimedia document occurring between t5 and tg, the representation 

11 of tiie multimedia information stored in the multimedia document occurring between U and tg 

12 displayed in the second area comprising a representation of information of the first type 

1 3 occurring between 15 and tg and a representation of information of the second type occurring 

14 between 15 and tg. 

1 96. (Currently amended) The system of claim 94 wherein the plurality of 

2 code modules fiirther comprises: 
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a code module for changing [[the]] position of the second lens in response to user 



4 input such that the second lens visually emphasizes a portion of the second area of the GUI 

5 comprising a representation of tiie multimedia information stored in the multimedia document 

6 occurring between a fifth time (t5) and a sixth time (tg), where (tj < t5 < tg < t2), (t5 ^ t3), and 

7 (tg ^ 14); and 



9 displaying, in the third area of the GUI, the representation of the multimedia information stored 

10 in the multimedia document occurring between t5 and t^, the representation of the multimedia 

1 1 information stored in the multimedia document occurring between t f and % displayed in the 

12 third area comprising a representation of information of the first type occurring between t5 and 

13 tg and a representation of information of the second type occurring between t5 and tg. 

1 97. (Currently amended) The system of claim 94 wherein the plurality of 

2 code modules further comprises: 

3 a code module for displaying , in the first area of the GUI, a third lens positionable 

4 over a plurality of positions within the first area of the GUI, the third lens visually emphasizing a 

5 portion of the first area of the GUI comprising a representation of the multimedia information 

6 stored in the multimedia document occurring between t3 and t4 

1 98. (Currently amended) The system of claim 97 wherein the plurality of 

2 code modules fiirther comprises: 

3 a code module for changing [[the]] position of the second lens in response to user 

4 input such that the second lens visually emphasizes a portion of the second area of the GUI 

5 comprising a representation of tiie multimedia information stored in the multimedia document 

6 occurring between a fifth time (t5) and a sixth time (tg), where (tj < t5 < tg < t2), (tf ^ t3), and 

7 (tg ^ t4); and 

8 responsive to the change in the position of the second lens, a code module for 

9 automatically changing [[the]] position of the third lens such that the third lens visually 



8 



responsive to the change in the position of the second lens, automatically 
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10 emphasizes a portion of the first area of the GUI comprising a representation of tiie multimedia 

1 1 information stored in the multimedia document occurring between t5 and tg 

1 99. (Currently amended) The system of claim 93 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises audio information; 

4 the representation of the information of the first type occurring between tg and t g 

5 comprises one or more video keyframes extracted from the video information; and 

6 the representation of information of the second type occurring between tg and tg 

7 comprises text information obtained from franscribing the audio information. 

1 100. (Currently amended) The system of claim 93 wherein: 

2 the information of the first type [[is]] comprises video information; 

3 the information of the second type [[is]] comprises closed-caption (CC) text 

4 information; 

5 the representation of the information of the first type occurring between tg andtg 

6 comprises one or more video keyframes extracted from the video information; and 

7 the representation of the information of the second type occurring between tg and 

8 te_comprises text information included in the CC text information. 

1 101 . (Currently amended) The system of claim 93 wherein the plurality of 

2 code modules further comprises: 

3 a code module for receiving information indicating a first topic; and 

4 a code module for analyzing the multimedia information stored in the multimedia 

5 document to identify one or more locations in the multimedia information that are relevant to the 

6 first topic; 

7 wherein the code module for displayin g, in the first area of the GUI, the 

8 representation of the multimedia information stored [[by]] in the multimedia document occurring 

9 between tg and tg in the first area of the GUI comprises highlighting the one or more locations in 
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10 the multimedia information displayed in the first area of the GUI that are relevant to the first 

1 1 topic ; and 

1 2 wherein the code module for displayin g, in the second area of the GUI, the 

13 representation of the multimedia information stored in the multimedia document occurring 

14 between tj and t2 in the second area of the GUI comprises highlighting the one or more locations 

15 in the multimedia information displayed in the second area of the GUI that are relevant to the 

1 6 first topic and that occur between times t j and t2 . 

1 102. (Original) The system of claim 93 wherein the plurality of code modules 

2 further comprises: 

3 a code module for receiving input indicating selection of a portion of the 

4 multimedia information occurring between a selection start time and a selection end time; and 

5 a code module for performing a first operation on the portion of the multimedia 

6 information occurring between the selection start time and the selection end time. 

1 103. (Currently amended) A system of displaying multimedia information 

2 stored in a multimedia document on a display, the muhim e dia information comprising vid e o 

3 information and information of a first typ e , the system comprising: 

4 a display; 

5 a processor; and 

6 a memory coupled to the processor, the memory configured to store a computer 

7 program; 

8 wherein the processor is operative with the computer program to: 

9 display a graphical user interface (GUI) on the display; 

1 0 display, in a first section of a first area of the GUI, a first set of one or 

1 1 more video keyfi-ames extracted fi-om [[the]] video information stored in the multimedia 

12 document occurring between a start time (tg) and an end time (tg) associated with the multimedia 

13 document, where (tg > t^); 



Page 52 of 69 



Appl. No. 10/081,129 PATENT 



Amdt. dated September 29, 2008 
Reply to Office Action of April 28, 2008 

14 display, n a second section of the first area of the GUI, text information 

1 5 corresponding to [[the]] information of [[the]] a first type stored in the multimedia document 

1 6 occurring between tg and t^; 

17 display a first lens movoablo in response to user input over roprosontations 

18 of multimedia information displayed in the GUI positionable over a plurality of portions of the 



19 first area of the GUI , the first lens emphasizing a portion of the first section of the first area 

20 occurring between a first time (tj) and a second time (t2) and a portion of the second section of 

2 1 the first area occurring between t j and t2, the emphasized portion of the first section of the first 

22 area comprising a second set of one or more video keyframes extracted from the video 

23 information occurring between tj and t2, the emphasized portion of the second section of the 

24 first area comprising text information corresponding to information of the first type occurring 

25 between t^ and t2, wherein the second set of one or more keyframes [[is]] comprises a subset of 

26 the first set of one or more keyframes and (tg < tj < t2 < tg); 



27 displa y, in a first section of a second area of the GUI, the second set of one 

28 or more keyframes based on the first lens emphasizing the portion of the first section of the first 

29 area in a first section of a second area of the GUI ; and 

30 displa y, in a second section of the second area of the GUT, text 

3 1 information corresponding to the information of the first type occurring between t^ and t2 based 

32 on the first lens emphasizing the portion of the second section of the first area in a second section 

33 of the second area of the GUI . 

1 104. (Currently amended) The system of claim 103 wherein the processor is 

2 operative with the computer program to : 

3 display a second lens movoablo in response to user input over roprosontations of 

4 multim e dia information display e d in th e GUI positionable over a plurality of portions of the 

5 second area of the GUI , the second lens emphasizing a portion of the first section of the second 

6 area and a portion of the second section of the second area, the emphasized portion of the first 

7 section of the second area comprising a third set of one or more video keyframes exfracted from 
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8 the video information occurring between a third time (t3) and a fourth time (t4), the emphasized 

9 portion of the second section of the second area comprising text information corresponding to 

1 0 information of the first type occurring between t3 and t/\, wherein the third set of one or more 

1 1 video keyframes [[is]] comprises a subset of the second set of one or more video keyfi'ames and 

12 (ti<t3<t4<t2); 

1 3 displa y, in a first section of a third area of the GUI, a keyframe from the third set 

14 of one or more keyframes based on the second lens emphasizing the portion of the first section of 

15 the second area in a first section of a third area of the GUI ; and 

16 displa y, in a second section of the third area of the GUI, text information 

1 7 corresponding to the information of the first type occurring between t3 and t4 based on the 

18 second lens emphasizing the portion of the second section of the second area in a second section 

19 of the third area of the GUI . 

1 105. (Currently amended) The system of claim 103 wherein the processor is 

2 operative with the computer program to: 

3 display a second lens positionablc over a plurality of portions within the second 

4 area of the GUI, the second lens emphasizing a portion of the first section of the second area and 

5 a portion of the second section of the second area, the emphasized portion of the first section of 

6 the second area comprising a third set of one or more video keyframes extracted from the video 

7 information occurring between a third time (t3) and a fourth time (t4), the emphasized portion of 

8 the second section of the second area comprising text information corresponding to information 

9 of the first type occurring between t3 and t4, wherein the third set of one or more video 

1 0 keyframes [[is]] comprises a subset of the second set of one or more video keyframes and (tj < 

11 t3<t4<t2); 

12 output video information starting from t3 or from t4 or from a time between 13 

1 3 and t4 in a first section of a third area of the GUI; and 
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14 display text information corresponding to the information of the first type 

1 5 occurring between 13 and in a second section of the third area of the GUI based on the first 

16 lens emphasizing the portion of the second section of the second area . 

1 106. (Currently amended) The system of claim 103 wherein the information of 

2 the first type between t ^ and t o [[is]] comprises audio information, and the text information 

3 corresponding to the information of the first type occurring between t ^ and t o is obtained from 

4 transcribing the audio information. 

1 107. (Currently amended) The method of claim 28 wherein the information of 

2 the first type between t ^ andt 2 [[is]] comprises closed-caption (CC) text information, and the 

3 text information corresponding to the information of the first type between t ^ and t o is extracted 

4 fi"om the CC text information. 

1 1 08. (Currently amended) The system of claim 1 03 wherein the multimedia 

2 information stored [[by]] in the multimedia document further comprises slides information, and 

3 wherein the processor is operative with the computer program to: 

4 display, in a third section of the first area of the GUI, a first set of one or more 

5 slides extracted from the slides information occurring between tg and tg, wherein the first lens 

6 emphasizes a portion of the third section of the first area comprising a second set of one or more 

7 slides extracted from the slides information occurring between t^ and t2, the second set of one or 

8 more slides [[is]] comprising a subset of the first set of one or more slides; and 

9 display the second set of one or more slides in a third section of the second area of 
1 0 the GUI based on the first lens emphasizing the portion of the third section of the first area . 

1 109. (Currently amended) The system of claim 108 wherein the processor is 

2 operative with the computer program to : 

3 display a second lens positionable over a plurality of portions of the second area. 

4 the second lens emphasizing a portion of the first section of the second area, a portion of the 
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5 second section of the second area, and a portion of the third section of the second area, the 

6 emphasized portion of the first section of the second area comprising a third set of one or more 

7 video keyframes extracted from the video information occurring between a third time (t3) and a 

8 fourth time (14), the emphasized portion of the second section of the second area comprising text 

9 information corresponding to information of the first type occurring between t3 and 14, the 

1 0 emphasized portion of the third section of the second area comprising a third set of one or more 

1 1 slides extracted from the slides information occurring between 13 and 14, wherein the third set of 

12 one or more video keyframes [[is]] comprises a subset of the second set of one or more video 

13 keyframes, the third set of one or more slides [[is]] comprising a subset of the second set of one 

14 or more slides, and (tj < t3 < t4 < t2); 

1 5 displa y, in a first section of a third area of the GUI, at least one keyfi-ame from the 

16 third set of one or more video keyframes based on the second lens emphasizing the portion of the 

17 first section of the second area in a first section of a third aroa of the GUI : 

1 8 displa y, in a second section of the third area of the GUI, the text information 

1 9 corresponding to the information of the first type occurring between t3 and t4 based on the 

20 second lens emphasizing the portion of the second section of the second area in a second section 

21 of the third area of the GUI ; and 

22 displa y, in a third section of the third area of the GUI, at least one slide from the 

23 third set of one or more slides based on the second lens emphasizing the portion of the third 

24 section of the second area in a third section of the third area of the GUI . 

1 110. (Currently amended) The system of claim 103 wherein the multimedia 

2 information stored [[by]] in the multimedia document further comprises whiteboard images 

3 information, and wherein the processor is operative with the computer program to: 

4 display, in a third section of the first area of the GUI, a first set of one or more 

5 whiteboard images extracted from the whiteboard images information occurring between tg and 

6 tg, wherein the first lens emphasizes a portion of the third section of the first area comprising a 

7 second set of one or more whiteboard images exfracted from the whiteboard images information 
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8 occurring between tj and t2, the second set of one or more whiteboard images [[is]] comprising a 

9 subset of the first set of one or more whiteboard images; and 

1 0 displa y, in a third section of the second area of the GUI, the second set of one or 

1 1 more whiteboard images based on the first lens emphasizing the portion of the third section of 

12 the first area in a third section of the second area of the GUI . 

1 111. (Currently amended) The system of claim 110 wherein the processor is 

2 operative with the computer program to: 

3 display a second lens positionable over a plurality of portions of the second area 

4 of the GUI, the second lens emphasizing a portion of the first section of the second area, a 

5 portion of the second section of the second area, and a portion of the third section of the second 

6 area, the emphasized portion of the first section of the second area comprising a third set of one 

7 or more video keyframes extracted from the video information occurring between a third time 

8 (t3) and a fourth time (t4), the emphasized portion of the second section of the second area 

9 comprising text information corresponding to information of the first type occurring between t3 

1 0 and t4, the emphasized portion of the third section of the second area comprising a third set of 

1 1 one or more whiteboard images extracted fi-om the whiteboard images information occurring 

12 between 13 and 14, wherein the third set of one or more video keyframes [[is]] comprises a subset 

1 3 of the second set of one or more video keyframes, the third set of one or more whiteboard images 

14 [[is]] comprising a subset of the second set of one or more whiteboard images, and (tj < t3 < t4 < 

15 t2); 



1 6 displa v. in a first section of the third area of the GUI, at least one keyframe from 

1 7 the third set of one or more video keyfi-ames based on the second lens emphasizing the portion of 

18 the first section of the second area in a first section of a third area of the GUI : 

1 9 displa y, in a second section of the third area of the GUI, the text information 

20 corresponding to the information of the first type occurring between 13 and 14 based on the 

21 second lens emphasizing the portion of the second section of the second area in a second section 

22 of the third area of the GUI ; and 
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23 displa y, in a third section of the third area of the GUI, a whiteboard image[[s]] 

24 from the third set of one or more whiteboard images based on the second lens emphasizing the 

25 portion of the third section of the second area in a third s e ction of th e third ar e a of th e GUI . 
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